{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "pdf-title"
    ]
   },
   "source": [
    "# Dropout\n",
    "Dropout [1] is a technique for regularizing neural networks by randomly setting some output activations to zero during the forward pass. In this exercise you will implement a dropout layer and modify your fully-connected network to optionally use dropout.\n",
    "\n",
    "[1] [Geoffrey E. Hinton et al, \"Improving neural networks by preventing co-adaptation of feature detectors\", arXiv 2012](https://arxiv.org/abs/1207.0580)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": [
     "pdf-ignore"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=========== You can safely ignore the message below if you are NOT working on ConvolutionalNetworks.ipynb ===========\n",
      "\tYou will need to compile a Cython extension for a portion of this assignment.\n",
      "\tThe instructions to do this will be given in a section of the notebook below.\n",
      "\tThere will be an option for Colab users and another for Jupyter (local) users.\n"
     ]
    }
   ],
   "source": [
    "# As usual, a bit of setup\n",
    "from __future__ import print_function\n",
    "import time\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from cs231n.classifiers.fc_net import *\n",
    "from cs231n.data_utils import get_CIFAR10_data\n",
    "from cs231n.gradient_check import eval_numerical_gradient, eval_numerical_gradient_array\n",
    "from cs231n.solver import Solver\n",
    "\n",
    "%matplotlib inline\n",
    "plt.rcParams['figure.figsize'] = (10.0, 8.0) # set default size of plots\n",
    "plt.rcParams['image.interpolation'] = 'nearest'\n",
    "plt.rcParams['image.cmap'] = 'gray'\n",
    "\n",
    "# for auto-reloading external modules\n",
    "# see http://stackoverflow.com/questions/1907993/autoreload-of-modules-in-ipython\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "def rel_error(x, y):\n",
    "  \"\"\" returns relative error \"\"\"\n",
    "  return np.max(np.abs(x - y) / (np.maximum(1e-8, np.abs(x) + np.abs(y))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": [
     "pdf-ignore"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X_train:  (49000, 3, 32, 32)\n",
      "y_train:  (49000,)\n",
      "X_val:  (1000, 3, 32, 32)\n",
      "y_val:  (1000,)\n",
      "X_test:  (1000, 3, 32, 32)\n",
      "y_test:  (1000,)\n"
     ]
    }
   ],
   "source": [
    "# Load the (preprocessed) CIFAR10 data.\n",
    "\n",
    "data = get_CIFAR10_data()\n",
    "for k, v in data.items():\n",
    "  print('%s: ' % k, v.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dropout forward pass\n",
    "In the file `cs231n/layers.py`, implement the forward pass for dropout. Since dropout behaves differently during training and testing, make sure to implement the operation for both modes.\n",
    "\n",
    "Once you have done so, run the cell below to test your implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running tests with p =  0.25\n",
      "Mean of input:  10.000207878477502\n",
      "Mean of train-time output:  2.5035147792443206\n",
      "Mean of test-time output:  10.000207878477502\n",
      "Fraction of train-time output set to zero:  0.749784\n",
      "Fraction of test-time output set to zero:  0.0\n",
      "\n",
      "Running tests with p =  0.4\n",
      "Mean of input:  10.000207878477502\n",
      "Mean of train-time output:  3.991167063504464\n",
      "Mean of test-time output:  10.000207878477502\n",
      "Fraction of train-time output set to zero:  0.600796\n",
      "Fraction of test-time output set to zero:  0.0\n",
      "\n",
      "Running tests with p =  0.7\n",
      "Mean of input:  10.000207878477502\n",
      "Mean of train-time output:  6.9914683385116\n",
      "Mean of test-time output:  10.000207878477502\n",
      "Fraction of train-time output set to zero:  0.30074\n",
      "Fraction of test-time output set to zero:  0.0\n",
      "\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(231)\n",
    "x = np.random.randn(500, 500) + 10\n",
    "\n",
    "for p in [0.25, 0.4, 0.7]:\n",
    "    out, _ = dropout_forward(x, {'mode': 'train', 'p': p})\n",
    "    out_test, _ = dropout_forward(x, {'mode': 'test', 'p': p})\n",
    "    \n",
    "    print('Running tests with p = ', p)\n",
    "    print('Mean of input: ', x.mean())\n",
    "    print('Mean of train-time output: ', out.mean())\n",
    "    print('Mean of test-time output: ', out_test.mean())\n",
    "    print('Fraction of train-time output set to zero: ', (out == 0).mean())\n",
    "    print('Fraction of test-time output set to zero: ', (out_test == 0).mean())\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dropout backward pass\n",
    "In the file `cs231n/layers.py`, implement the backward pass for dropout. After doing so, run the following cell to numerically gradient-check your implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dx relative error:  5.44560814873387e-11\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(231)\n",
    "x = np.random.randn(10, 10) + 10\n",
    "dout = np.random.randn(*x.shape)\n",
    "\n",
    "dropout_param = {'mode': 'train', 'p': 0.2, 'seed': 123}\n",
    "out, cache = dropout_forward(x, dropout_param)\n",
    "dx = dropout_backward(dout, cache)\n",
    "dx_num = eval_numerical_gradient_array(lambda xx: dropout_forward(xx, dropout_param)[0], x, dout)\n",
    "\n",
    "# Error should be around e-10 or less\n",
    "print('dx relative error: ', rel_error(dx, dx_num))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "pdf-inline"
    ]
   },
   "source": [
    "## Inline Question 1:\n",
    "What happens if we do not divide the values being passed through inverse dropout by `p` in the dropout layer? Why does that happen?\n",
    "\n",
    "## Answer:\n",
    "\n",
    "\n",
    "- dx数值变大了，但数量级不变。（为何数量级不变？\n",
    "- `Mean of train-time output`为`Mean of input`的`p`倍数（因为train过程只保留了比例为p的input，根据期望线性性，输出的均值应该为原来的p倍）\n",
    "- `Mean of test-time output`不变（因为invert dropout的test过程“无dropout”）\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fully-connected nets with Dropout\n",
    "In the file `cs231n/classifiers/fc_net.py`, modify your implementation to use dropout. Specifically, if the constructor of the network receives a value that is not 1 for the `dropout` parameter, then the net should add a dropout layer immediately after every ReLU nonlinearity. After doing so, run the following to numerically gradient-check your implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running check with dropout =  1\n",
      "Initial loss:  2.3004790897684924\n",
      "W1 relative error: 1.48e-07\n",
      "W2 relative error: 2.21e-05\n",
      "W3 relative error: 3.53e-07\n",
      "b1 relative error: 5.38e-09\n",
      "b2 relative error: 2.09e-09\n",
      "b3 relative error: 5.80e-11\n",
      "\n",
      "Running check with dropout =  0.75\n",
      "Initial loss:  2.302454105929313\n",
      "W1 relative error: 2.56e-07\n",
      "W2 relative error: 2.10e-06\n",
      "W3 relative error: 5.57e-08\n",
      "b1 relative error: 6.43e-09\n",
      "b2 relative error: 2.04e-09\n",
      "b3 relative error: 1.10e-10\n",
      "\n",
      "Running check with dropout =  0.5\n",
      "Initial loss:  2.3030067920900876\n",
      "W1 relative error: 1.22e-06\n",
      "W2 relative error: 9.41e-08\n",
      "W3 relative error: 2.67e-07\n",
      "b1 relative error: 3.53e-08\n",
      "b2 relative error: 3.37e-09\n",
      "b3 relative error: 1.01e-10\n",
      "\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(231)\n",
    "N, D, H1, H2, C = 2, 15, 20, 30, 10\n",
    "X = np.random.randn(N, D)\n",
    "y = np.random.randint(C, size=(N,))\n",
    "\n",
    "for dropout in [1, 0.75, 0.5]:\n",
    "  print('Running check with dropout = ', dropout)\n",
    "  model = FullyConnectedNet([H1, H2], input_dim=D, num_classes=C,\n",
    "                            weight_scale=5e-2, dtype=np.float64,\n",
    "                            dropout=dropout, seed=123)\n",
    "\n",
    "  loss, grads = model.loss(X, y)\n",
    "  print('Initial loss: ', loss)\n",
    "  \n",
    "  # Relative errors should be around e-6 or less; Note that it's fine\n",
    "  # if for dropout=1 you have W2 error be on the order of e-5.\n",
    "  for name in sorted(grads):\n",
    "    f = lambda _: model.loss(X, y)[0]\n",
    "    grad_num = eval_numerical_gradient(f, model.params[name], verbose=False, h=1e-5)\n",
    "    print('%s relative error: %.2e' % (name, rel_error(grad_num, grads[name])))\n",
    "  print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Regularization experiment\n",
    "As an experiment, we will train a pair of two-layer networks on 500 training examples: one will use no dropout, and one will use a keep probability of 0.25. We will then visualize the training and validation accuracies of the two networks over time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "(Iteration 1 / 125) loss: 7.856644\n",
      "(Epoch 0 / 25) train acc: 0.260000; val_acc: 0.184000\n",
      "(Epoch 1 / 25) train acc: 0.416000; val_acc: 0.258000\n",
      "(Epoch 2 / 25) train acc: 0.482000; val_acc: 0.276000\n",
      "(Epoch 3 / 25) train acc: 0.532000; val_acc: 0.277000\n",
      "(Epoch 4 / 25) train acc: 0.600000; val_acc: 0.271000\n",
      "(Epoch 5 / 25) train acc: 0.708000; val_acc: 0.299000\n",
      "(Epoch 6 / 25) train acc: 0.722000; val_acc: 0.282000\n",
      "(Epoch 7 / 25) train acc: 0.832000; val_acc: 0.255000\n",
      "(Epoch 8 / 25) train acc: 0.878000; val_acc: 0.269000\n",
      "(Epoch 9 / 25) train acc: 0.902000; val_acc: 0.275000\n",
      "(Epoch 10 / 25) train acc: 0.890000; val_acc: 0.260000\n",
      "(Epoch 11 / 25) train acc: 0.930000; val_acc: 0.281000\n",
      "(Epoch 12 / 25) train acc: 0.958000; val_acc: 0.301000\n",
      "(Epoch 13 / 25) train acc: 0.964000; val_acc: 0.305000\n",
      "(Epoch 14 / 25) train acc: 0.962000; val_acc: 0.314000\n",
      "(Epoch 15 / 25) train acc: 0.964000; val_acc: 0.302000\n",
      "(Epoch 16 / 25) train acc: 0.986000; val_acc: 0.305000\n",
      "(Epoch 17 / 25) train acc: 0.972000; val_acc: 0.318000\n",
      "(Epoch 18 / 25) train acc: 0.994000; val_acc: 0.312000\n",
      "(Epoch 19 / 25) train acc: 0.984000; val_acc: 0.313000\n",
      "(Epoch 20 / 25) train acc: 0.976000; val_acc: 0.309000\n",
      "(Iteration 101 / 125) loss: 0.021572\n",
      "(Epoch 21 / 25) train acc: 0.980000; val_acc: 0.318000\n",
      "(Epoch 22 / 25) train acc: 0.968000; val_acc: 0.312000\n",
      "(Epoch 23 / 25) train acc: 0.972000; val_acc: 0.309000\n",
      "(Epoch 24 / 25) train acc: 0.982000; val_acc: 0.306000\n",
      "(Epoch 25 / 25) train acc: 0.978000; val_acc: 0.305000\n",
      "\n",
      "0.2\n",
      "(Iteration 1 / 125) loss: 4.295453\n",
      "(Epoch 0 / 25) train acc: 0.244000; val_acc: 0.199000\n",
      "(Epoch 1 / 25) train acc: 0.382000; val_acc: 0.258000\n",
      "(Epoch 2 / 25) train acc: 0.402000; val_acc: 0.253000\n",
      "(Epoch 3 / 25) train acc: 0.500000; val_acc: 0.274000\n",
      "(Epoch 4 / 25) train acc: 0.548000; val_acc: 0.290000\n",
      "(Epoch 5 / 25) train acc: 0.604000; val_acc: 0.295000\n",
      "(Epoch 6 / 25) train acc: 0.650000; val_acc: 0.304000\n",
      "(Epoch 7 / 25) train acc: 0.660000; val_acc: 0.307000\n",
      "(Epoch 8 / 25) train acc: 0.708000; val_acc: 0.317000\n",
      "(Epoch 9 / 25) train acc: 0.708000; val_acc: 0.313000\n",
      "(Epoch 10 / 25) train acc: 0.752000; val_acc: 0.313000\n",
      "(Epoch 11 / 25) train acc: 0.776000; val_acc: 0.317000\n",
      "(Epoch 12 / 25) train acc: 0.744000; val_acc: 0.298000\n",
      "(Epoch 13 / 25) train acc: 0.802000; val_acc: 0.310000\n",
      "(Epoch 14 / 25) train acc: 0.804000; val_acc: 0.317000\n",
      "(Epoch 15 / 25) train acc: 0.822000; val_acc: 0.340000\n",
      "(Epoch 16 / 25) train acc: 0.862000; val_acc: 0.324000\n",
      "(Epoch 17 / 25) train acc: 0.858000; val_acc: 0.332000\n",
      "(Epoch 18 / 25) train acc: 0.846000; val_acc: 0.344000\n",
      "(Epoch 19 / 25) train acc: 0.870000; val_acc: 0.342000\n",
      "(Epoch 20 / 25) train acc: 0.878000; val_acc: 0.327000\n",
      "(Iteration 101 / 125) loss: 1.625189\n",
      "(Epoch 21 / 25) train acc: 0.896000; val_acc: 0.292000\n",
      "(Epoch 22 / 25) train acc: 0.892000; val_acc: 0.302000\n",
      "(Epoch 23 / 25) train acc: 0.902000; val_acc: 0.312000\n",
      "(Epoch 24 / 25) train acc: 0.916000; val_acc: 0.324000\n",
      "(Epoch 25 / 25) train acc: 0.914000; val_acc: 0.327000\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Train two identical nets, one with dropout and one without\n",
    "np.random.seed(231)\n",
    "num_train = 500\n",
    "small_data = {\n",
    "  'X_train': data['X_train'][:num_train],\n",
    "  'y_train': data['y_train'][:num_train],\n",
    "  'X_val': data['X_val'],\n",
    "  'y_val': data['y_val'],\n",
    "}\n",
    "\n",
    "solvers = {}\n",
    "dropout_choices = [1, 0.2]\n",
    "for dropout in dropout_choices:\n",
    "  model = FullyConnectedNet([500], dropout=dropout)\n",
    "  print(dropout)\n",
    "\n",
    "  solver = Solver(model, small_data,\n",
    "                  num_epochs=25, batch_size=100,\n",
    "                  update_rule='adam',\n",
    "                  optim_config={\n",
    "                    'learning_rate': 5e-4,\n",
    "                  },\n",
    "                  verbose=True, print_every=100)\n",
    "  solver.train()\n",
    "  solvers[dropout] = solver\n",
    "  print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA4IAAAJNCAYAAABkwMu0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzs3XucXXV56P/PYxhkQJrhEq3JBBMlDWLIMe0AnqK0mmJABCIqAhXEG2LFWNtGoT/NCTm1UmLFHwd+bVER8IZTD8agSFTQtnhBJoZOuEUCopkJSrgkgg6ShOf3x94T9kwmk5nM7L1m7/15v17z2rOetfZaz957zWSefG+RmUiSJEmSmsdzik5AkiRJklRbFoKSJEmS1GQsBCVJkiSpyVgISpIkSVKTsRCUJEmSpCZjIShJkiRJTcZCUJLUFCJiUkQ8GRGHFJ2LJElFC9cRlCRNRBHxZMXmvsDvge3l7fdk5hdrn5UkSY3BQlCSNOFFxIPAuzLzu8Mcs1dmbqtdVrXTyK9NklQMu4ZKkupSRPxDRHwlIr4cEU8Ab42I/xkRP46IzRHxUERcFhEt5eP3ioiMiBnl7S+U938rIp6IiB9FxMxdXOs5EfHViPhV+dzfj4iXVuzfNyIujYhfRsSWiPjPiHhued+x5Zy2RMSGiDirHL81Is6pOMe7IuL7g3L9q4hYD9xbjl8eET0R8ZuIuD0i/rTi+XtFxEcj4v7y/q6ImBoR/xYR/zTo9XwrIs4f+6cgSapXFoKSpHr2BuBLwGTgK8A24APAwcAxwPHAe4Z5/pnAR4EDgV8C/3uYY78BzAL+ELgT+HzFvkuBucDR5XP9PfBMubD8JvBJ4CBgHrB2FK/vZOBI4Ijy9m3l6xwIfBX49/6CE1gMvInSa24D3gU8BVwDnBkRzwGIiBcAfwZcN4o8JEkNxkJQklTPbs3MGzLzmczsy8zbM/O2zNyWmQ8AV1Iqenblq5nZlZlbgS8CLx/qoPL5r87MJzLzKWAp8CcRsV9ETALOARZl5kOZuT0zby2f863ATZnZWc7pkcy8YxSv7x8z8/HM7Cvn8fnMfKzcTfQS4A+AQ8vHvgv4+8y8r5zvHeVjfwj0VbwPZwDfzcxHRpGHJKnBWAhKkurZhsqNiDgsIr5Z7sL5G2AZpdbBXflVxfe/A5431EHlGUcviYgHyuddX951MPACYG/g/iGeOn0X8ZEa/Po+FBH3RsQW4HFgP559fcNd61pKRSnlx8/v4jhJUpOwEJQk1bPBM579G6Vum4dm5h8AS4AYh+ucDbwOeA2lbqj9rXAB/Bp4GnjJEM/bsIs4wG8pzYba7w+HOGbH64uIVwN/A7yRUtfPA4Anefb1DXetzwOnRsS88jE37OI4SVKTsBCUJDWS/YEtwG/Lk7kMNz5wtOf9PfAopeLtY/07MnM7cDXwqYj4w3Lr4THlSWq+ABwfEW8sT+ZycET8j/JT7wDeGBGtEfFHwDtGkMM24BGghVL31P0q9n8G+IeIeEmUvDwiDizn+Ivy9a4B/r3cvVWS1MQsBCVJjeRvgbcBT1BqHfzKOJ33c8DG8tddwA8H7f8gcA+wGngM+EdKSzT9HDgJ+HA5/lOenfjlE5Ra/B4GrqJUNA7nRuC7wH3Ag8BvgIcq9i8HVgA3l/ddCexTsf+a8rXtFipJch1BSZKaQUS8Bvgs8OL0H39Janq2CEqS1OAiYm9Ky2p82iJQkgQWgpIkNbSIOILSDKMHApcVnI4kaYKwa6gkSZIkNRlbBCVJkiSpyVgISpIkSVKT2avoBMbLwQcfnDNmzCg6DUmSJEkqxOrVqx/JzCkjObZhCsEZM2bQ1dVVdBqSJEmSVIiI+MVIj7VrqCRJkiQ1GQtBSZIkSWoyFoKSJEmS1GQKKQQj4qqIeDgi7tzF/oiIyyJifUR0R8Qf1zpHSZIkSWpURbUIXg0cP8z+E4BZ5a9zgX+pQU6SJEmS1BQKKQQz8z+Bx4Y55BTg2iz5MdAWES+sTXaSJEmS1Ngm6vIR04ANFds95dhDxaQjSZJUf1as6WX5qnVs3NzH1LZWFi+YzcJ504pOS9IEMFELwRgiljsdFHEupa6jHHLIIdXOSZIkqW6sWNPLhdevpW/rdgB6N/dx4fVrASwGJU3YWUN7gOkV2+3AxsEHZeaVmdmRmR1TpkypWXKSJKn+rFjTyzEX38LMC77JMRffwoo1vUWnVFXLV63bUQT269u6neWr1hWUkaSJZKK2CK4Ezo+I64CjgS2ZabdQSZK0R5qxdWzj5r5Rxceb3VKlia2QQjAivgz8OXBwRPQA/wtoAcjMfwVuBF4HrAd+B7y9iDwlSVJjGK51rFGLk6ltrfQOUfRNbWut+rWbsfAGi99a8r0eu0IKwcw8Yzf7E3hfjdKRJEkNrujWsSIsXjB7QDEG0NoyicULZlf92kUW3kUVCM1a/BahyPe6kQrQido1VJIkadwU2ToGxfzx2H/+Iv5oLarwLrJAaMZWZyjm3i7qvW60Yt9CUJIkNbwiW8eK/ONx4bxphfyBWlThXWQx1oytzkXd20W9141W7E/UWUMlSZLGzcJ50/j4qUcwra2VAKa1tfLxU4+oyR9vzTh75+IFs2ltmTQgVovCu8hibFdFbq3GZBYxI25R93ZR73WjFfu2CEqSVKAix5sUOZaqiOsW1TrWaH88jkRR3VKL7AJcVKtzkS3ORd3bRb3XRXcxH28WgpIkFaToCQ+KuHajjbEZiUb743Gkiii8i+wCXFTxW2R3xaLu7aLe6yLvr2qwEJQkieaa8KDIazfaGJuRaLQ/HieyIifI6b9+re/jIluciy68a/1eF31/jTcLQUlS02u2CQ+KvLbdJOv/j8eJrqguwEUpssW5Ge/tRrq/LAQlSU2vqFaqIv+AK+radpOUxlfRLc7e2/XLWUMlSU2vyAkPiphZschrF/mapUZU5Iy4qm+2CEqSml6zTXhQ5LWbsSuZVG22ymlPRGYWncO46OjoyK6urqLTkCTVocFjBKHUSuX/qkuS6klErM7MjpEca4ugJKnp2UolSWo2FoKSJGHXKklSc7EQlCRNGEWs5SdJUjOyEJQkTQhFreUnSVIzcvkISdKEMNxafpIkaXxZCEqSJoSi1vKTJKkZWQhKkiaEXa3ZV+21/CRJdai7Ey6dA0vbSo/dnY193SqwEJQkTQiLF8ymtWXSgFhryyQWL5hdUEaSpAmpuxNuWARbNgBZerxhUfWLsqKuWyUWgpKkCWHhvGl8/NQjmNbWSgDT2lpd0F2StLObl8HWQcMGtvaV4o143SopZNbQiDge+H+BScBnMvPiQftfBFwFTAEeA96amT01T1SSVFOu5SdJ2q0tuygLdhWv9+tWSc1bBCNiEnAFcAJwOHBGRBw+6LBPANdm5lxgGfDx2mYpScVbsaaXYy6+hZkXfJNjLr6FFWt6i05JkqSBihgzN7l9dPF6v26VFNE19ChgfWY+kJlPA9cBpww65nDg5vL33xtivyQ1tP419Xo395E8u6aexaAkacIoaszc/CXQMmgisZbWUrwRr1slRRSC04ANFds95Vil/wbeWP7+DcD+EXFQDXKTpAnBNfUkqQ410IySI1LUmLm5p8FJl8Hk6UCUHk+6rBRvxOtWSRFjBGOIWA7a/jvg8og4B/hPoBfYttOJIs4FzgU45JBDxjdLSSpQ0WvqrVjTy/JV69i4uY+pba0sXjDbsXuSNJz+1rH+wqi/dQyqXyh0d5aKry09pW6K85fUpjgpcszc3NOKKcCKum4VFNEi2ANMr9huBzZWHpCZGzPz1MycB/w/5diWwSfKzCszsyMzO6ZMmVLNnCWppopcU89uqZK0B4pqHStySYMGGzPXbIooBG8HZkXEzIjYGzgdWFl5QEQcHBH9uV1IaQZRSWoaRa6pZ7dUSdoDRbWOFbmkQYONmWs2NS8EM3MbcD6wCrgH6MzMuyJiWUScXD7sz4F1EfEz4AXAx2qdpyQVqcg19YrulipJdamo1rGiu2c20Ji5ZlPIOoKZeSNw46DYkorvvwp8tdZ5SdJQihovV9SaelPbWukdouirRbdUSRqzosbLzV8ycIwg1KZ1bHJ7uVvoEPFaaKAxc82miK6hklQ3mnG8XJHdUiVpTIocL1dU65jdM7WHInPwhJ31qaOjI7u6uopOQ1KDOebiW4ZsHZvW1soPLnhNARnVhrOGShqzIlrmLp2zi9ax6fDBO6t77SIV1QqqCSciVmdmx0iOLaRrqCTVi2YdL1dUt1RJDaKopRSKHC9XJLtnag/YNVSShlHkMg6SxlmzLfZdpKJmsnQ5A2nELAQlaRiOl5MaRJFjx5pRUS1zjpeTRsxCUJKGUeQyDpLGUZFrrUHztUYW1TLncgbSiDlGUJJ2w/FyUgMocuxYUePlilTUUgrgeDlphGwRlCRJja/IsWNFt0YWwZY5acKzRVCSJDW+Iluoim6NLGpZAVvmpAnNFkFJktT4imyhKqo10glyJA3DFkFJktQcimqhKqo1crguqbbUSU3PFkFJkqRqKqo1slkXV5c0IrYISqobK9b0snzVOjZu7mNqWyuLF8x2Nk9J9aGI1sjJ7eVuoUPEJTU9WwQl1YUVa3q58Pq19G7uI4HezX1ceP1aVqzpLTo1SZqYXFxd0jAsBCXVheWr1tG3dfuAWN/W7Sxfta6gjCRpgnMJB0nDsGuopLqwcXPfqOKSJrAilzRoNi7hIGkXbBGUVBemtrWOKi5pgnJJA0maECwEJdWFxQtm09oyaUCstWUSixfMLigjSXtkuCUNJEk1Y9dQSXWhf3ZQZw2V6pxLGkjShGAhKKluLJw3zcJPqncuaSBJE0IhXUMj4viIWBcR6yPigiH2HxIR34uINRHRHRGvKyJPSZIaWncnXDoHlraVHmsxTs8lDSRpQqh5IRgRk4ArgBOAw4EzIuLwQYd9BOjMzHnA6cD/V9ssJUmqkSKKsf7rFjFpi0saSNKEUETX0KOA9Zn5AEBEXAecAtxdcUwCf1D+fjKwsaYZSpJUC/3FWP/kKf3FGFS/MBpu0pZqX9slDSSpcEV0DZ0GVA4O6CnHKi0F3hoRPcCNwPtrk5okSTVU5AyaTtoiSU2tiEIwhojloO0zgKszsx14HfD5iNgp14g4NyK6IqJr06ZNVUhVkqQqKrIY29XkLE7aIklNoYhCsAeYXrHdzs5dP98JdAJk5o+AfYCDB58oM6/MzI7M7JgyZUqV0pUkqUqKLMactEWSmloRheDtwKyImBkRe1OaDGbloGN+CcwHiIiXUioEbfKTJDWWIosxJ22RpKZW88liMnNbRJwPrAImAVdl5l0RsQzoysyVwN8Cn46ID1LqNnpOZg7uPipJUn3rL7puXlbqDjq5vVQE1qoYc9IWSWpa0Sj1VUdHR3Z1dRWdhiRpLLo7iyuKJEmqcxGxOjM7RnJsEctHSJK0syKXUpAkqckUMUZQkqSdFbmUgiRJTcZCUJI0MbiunSRJNWMhKEmaGFzXTpKkmrEQlCRNDK5rJ0lSzVgISpImBte1kySpZpw1VJK0s6KWcXBdO0mSasJCUJI0kMs4SJLU8OwaKkkayGUcJElqeBaCkqSBmnUZh+5OuHQOLG0rPXZ3Fp2RJElVYyEoSRqoGZdx6O8Ou2UDkM92h7UYlCQ1KAtBSdJAzbiMg91hJUlNxkJQkjRQMy7j0KzdYSVJTctZQyVJO2u2ZRwmt5e7hQ4RlySpAdkiKElSM3aHlSQ1NVsEJY3KijW9LF+1jo2b+5ja1sriBbNZOG9a0WlJY9Pf+nnzslJ30MntpSKwmVpFJUlNxUJQ0oitWNPLhdevpW/rdgB6N/dx4fVrASwGVf+arTusJKmp2TVU0ogtX7VuRxHYr2/rdpavWldQRpIkSdoTFoKSRmzj5r5RxSVJkjQx2TVUqlNFjNWb2tZK7xBF39S21iGObiDdncWMHSvqupIkqeHZIijVof6xer2b+0ieHau3Yk1vVa+7eMFsWlsmDYi1tkxi8YLZVb1uobo74YZF5aUFsvR4w6JSvBGvK0mSmkIhhWBEHB8R6yJifURcMMT+SyPijvLXzyJicxF5ShNVUWP1Fs6bxsdPPYJpba0EMK2tlY+fekRjTxRz8zLYOqgVdGtfKd6I15UkSU2h5l1DI2IScAVwHNAD3B4RKzPz7v5jMvODFce/H5hX6zyliazIsXoL501r7MJvsC09o4vX+3UlSVJTKKJF8ChgfWY+kJlPA9cBpwxz/BnAl2uSmVQndjUmr+HH6hVhcvvo4vV+XUmS1BSKKASnARsqtnvKsZ1ExIuAmcAtNchLqhtNOVavKPOXQMugArultRRvxOtKkqSmUMSsoTFELHdx7OnAVzNz+1A7I+Jc4FyAQw45ZHyyk+pAf9fMWs8a2pT6Z+ms9eydRV1XkiQ1hcjcVQ1WpQtG/E9gaWYuKG9fCJCZHx/i2DXA+zLzh7s7b0dHR3Z1dY13upIkSZJUFyJidWZ2jOTYMXUNjYjzI+KAUT7tdmBWRMyMiL0ptfqtHOLcs4EDgB+NJUdJkiRJ0kBjHSP4h5Rm/ewsLwkxVLfPATJzG3A+sAq4B+jMzLsiYllEnFxx6BnAdVnrJktJkiRJanBj7hpaLv5eC7wd6AA6gc9m5v1jT2/k7BoqSZIkqZnVrGsoQLnF7lflr22UunN+NSIuGeu5JUmSJEnjb0yzhkbEIuBtwCPAZ4DFmbk1Ip4D3Ad8aOwpSpIkSZLG01iXjzgYODUzf1EZzMxnIuL1Yzy3JEmSJKkKxto19Ebgsf6NiNg/Io4GyMx7xnhuSRqouxMunQNL20qP3Z1FZyRJklSXxloI/gvwZMX2b8sxSRpf3Z1wwyLYsgHI0uMNiywGJUmS9sBYC8GoXN4hM59h7N1NJWlnNy+DrX0DY1v7SnFJkiSNylgLwQciYlFEtJS/PgA8MB6JSZqgiuqeuaVndHFJkiTt0lgLwfOAPwV6gR7gaODcsSYlaYIqsnvm5PbRxSVJkrRLYyoEM/PhzDw9M5+fmS/IzDMz8+HxSk7SBFNk98z5S6CldWCspbUUlyRJ0qiMdR3BfYB3Ai8D9umPZ+Y7xpiXpImoyO6Zc08rPd68rHS9ye2lIrA/LkmSpBEb68QunwfuBRYAy4C/BFw2QmpUk9vL3UKHiNfC3NMs/CRJksbBWMcIHpqZHwV+m5nXACcCR4w9LUkTkt0zJUmSGsJYC8Gt5cfNETEHmAzMGOM5JU1Uc0+Dky6DydOBKD2edJmtdJIkSXVmrF1Dr4yIA4CPACuB5wEfHXNWkiYuu2dKkiTVvT0uBCPiOcBvMvNx4D+BF49bVpIkSZKkqtnjrqGZ+Qxw/jjmIkmSJEmqgbGOEfxORPxdREyPiAP7v8YlM0mSJElSVYx1jGD/eoHvq4gldhOVJEmSpAlrTIVgZs4cr0QkSZIkSbUxpkIwIs4eKp6Z147lvJJGoLsTbl4GW3pKC7rPX+JsnpIkSRqRsY4RPLLi61XAUuDk3T0pIo6PiHURsT4iLtjFMadFxN0RcVdEfGmMeUqNpbsTblgEWzYAWXq8YVEpLkmSJO3GWLuGvr9yOyImA58f7jkRMQm4AjgO6AFuj4iVmXl3xTGzgAuBYzLz8Yh4/ljylKplxZpelq9ax8bNfUxta2XxgtksnDet+he+eRls7RsY29pXitsqKEmSpN0Ya4vgYL8DZu3mmKOA9Zn5QGY+DVwHnDLomHcDV5TXKCQzHx7nPKUxW7GmlwuvX0vv5j4S6N3cx4XXr2XFmt7qX3xLz+jikiRJUoUxFYIRcUNErCx/fQNYB3x9N0+bBmyo2O4pxyr9EfBHEfGDiPhxRBw/ljylali+ah19W7cPiPVt3c7yVeuqf/HJ7aOLS5IkSRXGunzEJyq+3wb8IjN31yQRQ8Ry0PZelFoW/xxoB/4rIuZk5uYBJ4o4FzgX4JBDDhlF2tLYbdzcN6r4uJq/pDQmsLJ7aEtrKS5JkiTtxli7hv4SuC0z/yMzfwA8GhEzdvOcHmB6xXY7sHGIY76emVsz8+eUWhp36nKamVdmZkdmdkyZMmVPX4O0R6a2tY4qPq7mngYnXQaTpwNRejzpMscHSpIkaUTGWgj+O/BMxfb2cmw4twOzImJmROwNnA6sHHTMCuDVABFxMKWuog+MMVdpXC1eMJvWlkkDYq0tk1i8YHZtEph7GnzwTli6ufRoEShJkqQRGmvX0L3KE74AkJlPl4u7XcrMbRFxPrAKmARclZl3RcQyoCszV5b3vTYi7qZUXC7OzEfHmKs0rvpnBy1k1lBJkiRpDCJz8PC8UTw54jvA/ykXb0TEKcCizJw/TvmNWEdHR3Z1ddX6spIkSZI0IUTE6szsGMmxY20RPA/4YkRcXt7uAc4e4zklSZIkSVU01gXl7wdeERHPo9S6+MT4pCVJkiRJqpaxriP4jxHRlplPZuYTEXFARPzDeCUnSZIkSRp/Y5019ITKtf0y83HgdWM8pyRJkiSpisZaCE6KiOf2b0REK/DcYY6XJEmSJBVsrJPFfAG4OSI+V95+O3DNGM8pSZIkSaqisU4Wc0lEdAN/AQRwE/Ci8UhMkiRJklQdY+0aCvAr4BngjcB84J5xOKdUH7o74dI5sLSt9NjdWXRGkiRJ0m7tUYtgRPwRcDpwBvAo8BVKy0e8ehxzkya27k64YRFs7Sttb9lQ2gaYe1pxeUmSJEm7sactgvdSav07KTNfmZn/B9g+fmlJdeDmZc8Wgf229pXikiRJ0gS2p4XgGyl1Cf1eRHw6IuZTGiMoNY8tPaOLS5IkSRPEHnUNzcyvAV+LiP2AhcAHgRdExL8AX8vMb49jjtJurVjTy/JV69i4uY+pba0sXjCbhfOmVfeik9tL3UGHikuSJEkT2Jgmi8nM32bmFzPz9UA7cAdwwbhkJo3QijW9XHj9Wno395FA7+Y+Lrx+LSvW9Fb3wvOXQEvrwFhLaykuSZIkTWDjMWsoAJn5WGb+W2a+ZrzOKY3E8lXr6Ns6cIhq39btLF+1rroXnnsanHQZTJ4OROnxpMucKEaSJEkT3lgXlJcKt3Fz36ji42ruaRZ+kiRJqjvj1iIoFWVqW+uo4pIkSVKzsxBU3Vu8YDatLZMGxFpbJrF4weyCMpIkSZImNruGqu71zw5a81lDJUmSpDplIaiGsHDeNAs/SZIkaYTsGipJkiRJTaaQQjAijo+IdRGxPiJ2WncwIs6JiE0RcUf5611F5ClJkiRJjajmXUMjYhJwBXAc0APcHhErM/PuQYd+JTPPr3V+qlPdnXDzMtjSA5PbS4u6u6yDJEmSNKQiWgSPAtZn5gOZ+TRwHXBKAXmoUXR3wg2LYMsGIEuPNywqxSVJkiTtpIhCcBqwoWK7pxwb7I0R0R0RX42I6bVJTXXp5mWwddDi8Vv7SnFJkiRJOymiEIwhYjlo+wZgRmbOBb4LXDPkiSLOjYiuiOjatGnTOKepurGlZ3RxSZIkqckVUQj2AJUtfO3AxsoDMvPRzPx9efPTwJ8MdaLMvDIzOzKzY8qUKVVJVnVgcvvo4pIkSVKTK6IQvB2YFREzI2Jv4HRgZeUBEfHCis2TgXtqmJ/20Io1vRxz8S3MvOCbHHPxLaxY01ubC89fAi2tA2MtraW4JEmSpJ3UfNbQzNwWEecDq4BJwFWZeVdELAO6MnMlsCgiTga2AY8B59Q6T43OijW9XHj9Wvq2bgegd3MfF16/FqD6C733zw7qrKGSJEnSiETm4OF59amjoyO7urqKTqNpHXPxLfRu7tspPq2tlR9c8JoCMpIkSZKaS0SszsyOkRxbyILyajwbhygCh4tLkiRJKo6FoMbF1LbWUcUlSZIkFcdCUONi8YLZtLZMGhBrbZnE4gWzC8pIkiRJ0q7UfLIYNab+CWGWr1rHxs19TG1rZfGC2dWfKEaSJEnSqFkIatwsnDfNwk+SJEmqA3YNlSRJkqQmYyEoSZIkSU3GQlCSJEmSmoyFoCRJkiQ1GQtBSZIkSWoyFoKSJEmS1GQsBCVJkiSpyVgISpIkSVKTsRCUJEmSpCZjIShJkiRJTcZCUJIkSZKazF5FJ6Dxt2JNL8tXrWPj5j6mtrWyeMFsFs6bVnRakiRJkiYIC8EGs2JNLxdev5a+rdsB6N3cx4XXrwWwGJQkSZIE2DW04SxftW5HEdivb+t2lq9aV1BGkiRJkiYaC8EGs3Fz36jikiRJkppPIYVgRBwfEesiYn1EXDDMcW+KiIyIjlrmV8+mtrWOKj6uujvh0jmwtK302N1Z/WtKkiRJGrWaF4IRMQm4AjgBOBw4IyIOH+K4/YFFwG21zbC+LV4wm9aWSQNirS2TWLxgdnUv3N0JNyyCLRuALD3esMhiUJIkSZqAimgRPApYn5kPZObTwHXAKUMc97+BS4CnaplcvVs4bxofP/UIprW1EsC0tlY+fuoR1Z8o5uZlsHVQ99OtfaW4JEmSpAmliFlDpwEbKrZ7gKMrD4iIecD0zPxGRPxdLZNrBAvnTav9DKFbekYXlyRJklSYIloEY4hY7tgZ8RzgUuBvd3uiiHMjoisiujZt2jSOKWrUJrePLi5JkiSpMEUUgj3A9IrtdmBjxfb+wBzg+xHxIPAKYOVQE8Zk5pWZ2ZGZHVOmTKliytqt+UugZdCENC2tpbgkSZKkCaWIQvB2YFZEzIyIvYHTgZX9OzNzS2YenJkzMnMG8GPg5MzsKiDX+lTE7J1zT4OTLoPJ04EoPZ50WSkuSZIkaUKp+RjBzNwWEecDq4BJwFWZeVdELAO6MnPl8GfQsPpn7+yfuKV/9k6oflE29zQLP0mSJKkORGbu/qg60NHRkV1dNhpy6ZzyEg6DTJ4OH7yz9vlIkiRJqomIWJ2ZI1qDvZAF5VVFzt4pSZIkaTcsBBuNs3dKkiRJ2g0LwUbj7J2SJEmSdsNCsNE4e6ckSZKk3aj5rKGqAWfvlCRJkjQMWwQlSZIkqclYCEqSJElSk7EQlCRJkqQmYyEoSZIkSU3GQlCSJEmSmoyFoCRJkiQ1GQtBSZIkSWoyFoKSJEmS1GQsBCVJkiSpyVgISpIkSVKTsRCUJEmSpCZjIShJkiRJTcZCUJIkSZKajIWgJEmSJDW8ka4eAAAgAElEQVQZC0FJkiRJajKFFIIRcXxErIuI9RFxwRD7z4uItRFxR0TcGhGHF5GnJEmSJDWimheCETEJuAI4ATgcOGOIQu9LmXlEZr4cuAT4ZI3TlCRJkqSGVUSL4FHA+sx8IDOfBq4DTqk8IDN/U7G5H5A1zE+SJEmSGtpeBVxzGrChYrsHOHrwQRHxPuBvgL2B19QmNUmSJElqfEW0CMYQsZ1a/DLzisx8CfBh4CNDniji3IjoioiuTZs2jXOakiRJktSYiigEe4DpFdvtwMZhjr8OWDjUjsy8MjM7MrNjypQp45iiJEmSJDWuIgrB24FZETEzIvYGTgdWVh4QEbMqNk8E7qthfpIkSZLU0Go+RjAzt0XE+cAqYBJwVWbeFRHLgK7MXAmcHxF/AWwFHgfeVus8JUmSJKlRFTFZDJl5I3DjoNiSiu8/UPOkJEmSVFVbt26lp6eHp556quhUpLq2zz770N7eTktLyx6fo5BCUJIkSc2np6eH/fffnxkzZhAx1PyBknYnM3n00Ufp6elh5syZe3yeIsYISpIkqQk99dRTHHTQQRaB0hhEBAcddNCYW9YtBCVJklQzFoHS2I3Hz5GFYJWsWNPLMRffwswLvskxF9/CijW9RackSZLU9N7xjnfw/Oc/nzlz5uzymMxk0aJFHHroocydO5ef/vSnO/Zdc801zJo1i1mzZnHNNdfs9nrf//73ef3rXz8uue+Jq6++mo0bh1uprb7cdNNNzJ49m0MPPZSLL754yGM++clPcvjhhzN37lzmz5/PL37xix37/PyeZSFYBSvW9HLh9Wvp3dxHAr2b+7jw+rUWg5IkSQU755xzuOmmm4Y95lvf+hb33Xcf9913H1deeSXvfe97AXjssce46KKLuO222/jJT37CRRddxOOPP75HeWzbtm2PnjdajVQIbt++nfe9731861vf4u677+bLX/4yd999907HzZs3j66uLrq7u3nTm97Ehz70IcDPbzALwSpYvmodfVu3D4j1bd3O8lXrCspIkiSp/lSjh9Wxxx7LgQceOOwxX//61zn77LOJCF7xilewefNmHnroIVatWsVxxx3HgQceyAEHHMBxxx03ZFF50003cdhhh/HKV76S66+/fkd86dKlnHvuubz2ta/l7LPP5qmnnuLtb387RxxxBPPmzeN73/seUPrj/5RTTuH4449n9uzZXHTRRTvO8clPfpI5c+YwZ84cPvWpTwHw4IMPDmjh/MQnPsHSpUv56le/SldXF3/5l3/Jy1/+cvr6+sb03o1adydcOgeWtpUeuzvHdLqf/OQnHHroobz4xS9m77335vTTT+frX//6Tse9+tWvZt999wXgFa94BT09PQB+foM4a2gVbNw89Ie0q7gkSZIG6u9h1f+f6/09rAAWzptW1Wv39vYyffr0Hdvt7e309vbuMl7pqaee4t3vfje33HILhx56KG95y1sG7F+9ejW33norra2t/PM//zMAa9eu5d577+W1r30tP/vZz4BS0XPnnXey7777cuSRR3LiiScSEXzuc5/jtttuIzM5+uij+bM/+zMOOOCAIV/Hm970Ji6//HI+8YlP0NHRMS7vzYh1d8INi2Br+e/fLRtK2wBzT9ujUw71/t92223DPuezn/0sJ5xwwi6f38yfny2CVTC1rXVUcUmSJA1UZA+rzNwpFhG7jFe69957mTlzJrNmzSIieOtb3zpg/8knn0xra+lvwltvvZWzzjoLgMMOO4wXvehFOwqJ4447joMOOojW1lZOPfVUbr31Vm699Vbe8IY3sN9++/G85z2PU089lf/6r/8al9c87m5e9mwR2G9rXym+h0by/lf6whe+QFdXF4sXLx7x85vp87MQrILFC2bzpr1/yK17L+KB557JrXsv4k17/5DFC2YXnZokSVJdKLKHVXt7Oxs2bNix3dPTw9SpU3cZH2y44mS//fbb8f1QhcmuzrGrQhRgr7324plnntmxPdZlBcbFlp7RxUdgpO8/wHe/+10+9rGPsXLlSp773OeO6vnN8vlZCFbBwkk/4OKWz9D+nEd4TkD7cx7h4pbPsHDSD4pOTZIkqS4U2cPq5JNP5tprryUz+fGPf8zkyZN54QtfyIIFC/j2t7/N448/zuOPP863v/1tFixYMOC5hx12GD//+c+5//77Afjyl7+8y+sce+yxfPGLXwTgZz/7Gb/85S+ZPbvUcPCd73yHxx57jL6+PlasWMExxxzDsccey4oVK/jd737Hb3/7W772ta/xqle9ihe84AU8/PDDPProo/z+97/nG9/4xo5r7L///jzxxBPj/Rbt3uT20cVH4Mgjj+S+++7j5z//OU8//TTXXXcdJ5988k7HrVmzhve85z2sXLmS5z//+Tvifn4DOUawGm5exl7bB1bye21/qtQUvod9oiVJkprJ4gWzB4wRBGhtmTTmHlZnnHEG3//+93nkkUdob2/noosu4p3vfCf/+q//CsB5553H6173Om688UYOPfRQ9t13Xz73uc8BcOCBB/LRj36UI488EoAlS5bsNPHMPvvsw5VXXsmJJ57IwQcfzCtf+UruvPPOIXP5q7/6K8477zyOOOII9tprL66++uodrVevfOUrOeuss1i/fj1nnnnmjjFi55xzDkcddRQA73rXu5g3b96OXI4++mhmzpzJYYcdtuMa55xzDueddx6tra386Ec/2tGtsermLxk4RhCgpbUU30N77bUXl19+OQsWLGD79u284x3v4GUvexlQev0dHR2cfPLJLF68mCeffJI3v/nNABxyyCGsXLnSz2+QGK5Js550dHRkV1dX0WmULG0DhnpfA5ZurnU2kiRJE8I999zDS1/60hEfv2JNL8tXrWPj5j6mtrWyeMHsqk8UMxFcffXVdHV1cfnllxedyth0d5YaQrb0lFoC5y9pikaRWn1+Q/08RcTqzBzRzDK2CFbD5PbSzEhDxSVJkjQiC+dNa4rCr2HNPa0pCr96ZYtgNQyeLhdKTeEnXeYPgyRJalqjbRGUtGtjbRF0sphqmHtaqeibPB2I0qNFoCRJkqQJwq6h1WJTuCRJ0k4yc9jp+SXt3nj06rRFUJIkSTWxzz778Oijj47LH7FSs8pMHn30UfbZZ58xnccWQUmSJNVEe3s7PT09bNq0qehUpLq2zz770N4+tokoLQQlSZJUEy0tLcycObPoNCRh11BJkiRJajoWgpIkSZLUZCwEJUmSJKnJNMyC8hGxCfhF0XkM4WDgkaKTUEPzHlM1eX+pmry/VE3eX6qmiXp/vSgzp4zkwIYpBCeqiOjKzI6i81Dj8h5TNXl/qZq8v1RN3l+qpka4v+waKkmSJElNxkJQkiRJkpqMhWD1XVl0Amp43mOqJu8vVZP3l6rJ+0vVVPf3l2MEJUmSJKnJ2CIoSZIkSU3GQrCKIuL4iFgXEesj4oKi81FjiYgHI2JtRNwREV1F56P6FxFXRcTDEXFnRezAiPhORNxXfjygyBxVv3Zxfy2NiN7y77E7IuJ1Reao+hUR0yPiexFxT0TcFREfKMf9HaYxG+b+quvfYXYNrZKImAT8DDgO6AFuB87IzLsLTUwNIyIeBDoycyKuYaM6FBHHAk8C12bmnHLsEuCxzLy4/B9aB2Tmh4vMU/VpF/fXUuDJzPxEkbmp/kXEC4EXZuZPI2J/YDWwEDgHf4dpjIa5v06jjn+H2SJYPUcB6zPzgcx8GrgOOKXgnCRplzLzP4HHBoVPAa4pf38NpX/4pFHbxf0ljYvMfCgzf1r+/gngHmAa/g7TOBjm/qprFoLVMw3YULHdQwPcMJpQEvh2RKyOiHOLTkYN6wWZ+RCU/iEEnl9wPmo850dEd7nrqN32NGYRMQOYB9yGv8M0zgbdX1DHv8MsBKsnhojZD1fj6ZjM/GPgBOB95W5XklRP/gV4CfBy4CHgn4tNR/UuIp4H/F/grzPzN0Xno8YyxP1V17/DLASrpweYXrHdDmwsKBc1oMzcWH58GPgape7I0nj7dXlsRP8YiYcLzkcNJDN/nZnbM/MZ4NP4e0xjEBEtlP5I/2JmXl8O+ztM42Ko+6vef4dZCFbP7cCsiJgZEXsDpwMrC85JDSIi9isPViYi9gNeC9w5/LOkPbISeFv5+7cBXy8wFzWY/j/Qy96Av8e0hyIigM8C92TmJyt2+TtMY7ar+6vef4c5a2gVlaeQ/RQwCbgqMz9WcEpqEBHxYkqtgAB7AV/y/tJYRcSXgT8HDgZ+DfwvYAXQCRwC/BJ4c2Y64YdGbRf3159T6lKVwIPAe/rHc0mjERGvBP4LWAs8Uw7/PaVxXP4O05gMc3+dQR3/DrMQlCRJkqQmY9dQSZIkSWoyFoKSJEmS1GQsBCVJkiSpyVgISpIkSVKTsRCUJEmSpCZjIShJ0iARsT0i7qj4umAczz0jIupqrSlJUuPZq+gEJEmagPoy8+VFJyFJUrXYIihJ0ghFxIMR8U8R8ZPy16Hl+Isi4uaI6C4/HlKOvyAivhYR/13++tPyqSZFxKcj4q6I+HZEtBb2oiRJTclCUJKknbUO6hr6lop9v8nMo4DLgU+VY5cD12bmXOCLwGXl+GXAf2Tm/wD+GLirHJ8FXJGZLwM2A2+s8uuRJGmAyMyic5AkaUKJiCcz83lDxB8EXpOZD0REC/CrzDwoIh4BXpiZW8vxhzLz4IjYBLRn5u8rzjED+E5mzipvfxhoycx/qP4rkySpxBZBSZJGJ3fx/a6OGcrvK77fjmP2JUk1ZiEoSdLovKXi8Ufl738InF7+/i+BW8vf3wy8FyAiJkXEH9QqSUmShuP/QEqStLPWiLijYvumzOxfQuK5EXEbpf9MPaMcWwRcFRGLgU3A28vxDwBXRsQ7KbX8vRd4qOrZS5K0G44RlCRphMpjBDsy85Gic5EkaSzsGipJkiRJTcYWQUmSJElqMrYISpKaSkTMiIiMCMfJS5KaloWgJKmuRMSqiFg2RPyUiPiVBZ4kSbtnIShJqjdXA2dFRAyKnwV8MTO31T6l8REl/tssSao6/7GRJNWbFcCBwKv6AxFxAPB64Nry9okRsSYifhMRGyJi6UhPHhEXRMT9EfFERNwdEW8YtP/dEXFPxf4/LsenR8T1EbEpIh6NiMvL8aUR8YWK5w/omhoR34+Ij0XED4DfAS+OiLdXXOOBiHjPoBxOiYg7yq/v/og4PiLeHBGrBx33txGxYqSvXZLUPCwEJUl1JTP7gE7g7IrwacC9mfnf5e3flve3AScC742IhSO8xP2UiszJwEXAFyLihQAR8WZgafncfwCcDDwaEZOAbwC/AGYA04DrRvGyzgLOBfYvn+NhSoXtH1Bak/DSioLzKEoF7+Ly6zsWeBBYCcyMiJdWnPetwOdHkYckqUlYCEqS6tE1wJsjorW8fXY5BkBmfj8z12bmM5nZDXwZ+LORnDgz/z0zN5af+xXgPuCo8u53AZdk5u1Zsj4zf1HePxVYnJm/zcynMvPWUbyeqzPzrszclplbM/ObmXl/+Rr/AXybZ1tA3wlclZnfKefYm5n3Zubvga9QKv6IiJdRKkq/MYo8JElNwkJQklR3ykXWJuCUiHgxcCTwpf79EXF0RHyv3E1zC3AecPBIzh0RZ5e7XW6OiM3AnIrnTqfUYjjYdOAXYxifuGFQDidExI8j4rFyDq8bQQ5QKobPLI+fPAvoLBeIkiQNYCEoSapX11JqCTwL+HZm/rpi35codZWcnpmTgX8FBk8us5OIeBHwaeB84KDMbAPurHjuBuAlQzx1A3DILmYs/S2wb8X2Hw5xzI5FfSPiucD/BT4BvKCcw40jyIHM/DHwNKXWwzOxW6gkaRcsBCVJ9epa4C+Ad1PRLbRsf+CxzHyqPKbuzBGecz9KRdkmgIh4O6UWwX6fAf4uIv6kPMPnoeXi8SfAQ8DFEbFfROwTEceUn3MHcGxEHBIRk4ELd5PD3sBzyzlsi4gTgNdW7P8s8PaImB8Rz4mIaRFxWMX+a4HLgW2j7J4qSWoiFoKSpLqUmQ8CP6RUvK0ctPuvgGUR8QSwhNLkMiM5593APwM/An4NHAH8oGL/vwMfo9Ti+ATlGUwzcztwEnAo8EugB3hL+TnfoTR2rxtYzW7G7GXmE8Cics6PUypiV1bs/wnlCWSALcB/AC+qOMXnKRWvtgZKknYpMnP3R0mSpLpQnkDnYeCPM/O+ovORJE1MtghKktRY3gvcbhEoSRrOUIPaJUlSHYqIBylNKjPSNRMlSU3KrqGSJEmS1GTsGipJkiRJTcZCUJIkSZKaTMOMETz44INzxowZRachSZIkSYVYvXr1I5k5ZSTHNkwhOGPGDLq6uopOQ5IkSZIKERG/GOmxdg2VJEmSpCZjIShJkiRJTcZCUJIkSZKajIWgJEmSJDUZC0FJkiRJajIWgpIkSZLUZCwEJUmSGlV3J1w6B5a2lR67O4vOSNIE0TDrCEqSJKlCdyfcsAi29pW2t2wobQPMPa24vCRNCLYISpIkNaKblz1bBPbb2leKS2p6FoKSJEmNaEvP6OLjzW6p0oRW1UIwIo6PiHURsT4iLhhi/3kRsTYi7oiIWyPi8EH7D4mIJyPi76qZpyRJUsOZ3D66+Hjq75a6ZQOQz3ZLtRiUJoyqFYIRMQm4AjgBOBw4Y3ChB3wpM4/IzJcDlwCfHLT/UuBb1cpRkiQVwJai2pi/BFpaB8ZaWkvxarNbqjThVXOymKOA9Zn5AEBEXAecAtzdf0Bm/qbi+P2A7N+IiIXAA8Bvq5ijJEmqJScwqZ3+9/PmZaXuoJPbS0VgLd7norulStqtahaC04ANFds9wNGDD4qI9wF/A+wNvKYc2w/4MHAcYLdQSZIaxXAtRRaC42/uacW8r5Pby91Ch4hLmhCqOUYwhojlToHMKzLzJZQKv4+UwxcBl2bmk8NeIOLciOiKiK5NmzaNOWFJklRlthQ1hyK7pUoakWoWgj3A9IrtdmDjMMdfBywsf380cElEPAj8NfD3EXH+4Cdk5pWZ2ZGZHVOmTBmfrCVJUvUUOYGJamfuaXDSZTB5OhClx5Mus9VXmkCq2TX0dmBWRMwEeoHTgTMrD4iIWZl5X3nzROA+gMx8VcUxS4EnM/PyKuYqSZJqYf6SgWMEwZaiRlVUt1RJI1K1QjAzt5Vb8VYBk4CrMvOuiFgGdGXmSuD8iPgLYCvwOPC2auUjSZImgCInMJEk7RCZOw3bq0sdHR3Z1dVVdBqSJEmSVIiIWJ2ZHSM5tqoLykuSJEmSJh4LQUmSJElqMhaCkiRJktRkLAQlSZKqrbsTLp0DS9tKj92dRWckqclVc/kISZIkdXcOXDJjy4bSNjhbqqTC2CIoSZJUTTcvG7huIpS2b15WTD6ShIWgJElSdW3pGV1ckmrAQlCSJKmaJrePLi5JNWAhKEmSVE3zl0BL68BYS2spLkkFsRCUJEmqprmnwUmXweTpQJQeT7rMiWIkFcpZQyVJkqpt7mkWfrXS3VmaiGdLT6n77fwlvvcaPw10f1kISpIkqTG4VIeqqcHuL7uGSpIkqTG4VIeqqcHuLwtBSZIkNQaX6lA1Ndj9ZSEoSZKaQ3cnXDoHlraVHrs7i85I482lOppHET/PDXZ/WQhKkqTG1z+2Z8sGIJ8d22Mx2FhcqqM5FPXz3GD3l4WgJElqfA02tke74FIdzaGon+cGu7+cNVSSJDW+Bhvbo2G4VEfjK/LnuYHuL1sEJUlS42uwsT1SU/PneVxUtRCMiOMjYl1ErI+IC4bYf15ErI2IOyLi1og4vBw/LiJWl/etjojXVDNPSZLU4BpsbI/U1Px5HhdVKwQjYhJwBXACcDhwRn+hV+FLmXlEZr4cuAT4ZDn+CHBSZh4BvA34fLXylCRJTaDBxvZITc2f53FRzTGCRwHrM/MBgIi4DjgFuLv/gMz8TcXx+wFZjq+piN8F7BMRz83M31cxX0mS1MgaaGyP1PT8eR6zahaC04ANFds9wNGDD4qI9wF/A+wNDNUF9I3AGotASZJUr1as6WX5qnVs3NzH1LZWFi+YzcJ504pOS+PMz1n1pJpjBGOIWO4UyLwiM18CfBj4yIATRLwM+CfgPUNeIOLciOiKiK5NmzaNQ8qSJEnja8WaXi68fi29m/tIoHdzHxdev5YVa3qLTk3jyM9Z9aaahWAPML1iux3YOMzx1wEL+zcioh34GnB2Zt4/1BMy88rM7MjMjilTpoxDypI0hO5OuHQOLG0rPboAdfX4XteW73dNLF+1jr6t2wfE+rZuZ/mqdQVlpGrwc1a9qWbX0NuBWRExE+gFTgfOrDwgImZl5n3lzROB+8rxNuCbwIWZ+YMq5ihJw+vuhBsWPbtw7ZYNpW1wbMJ4872uLd/vmtm4uW9UcdUnP2fVm6q1CGbmNuB8YBVwD9CZmXdFxLKIOLl82PkRcVdE3EFpnODb+uPAocBHy0tL3BERz69WrpK0Szcve/YP5X5b+0pxjS/f69ry/a6ZqW2to4qrPvk5q95UdR3BzLwxM/8oM1+SmR8rx5Zk5sry9x/IzJdl5ssz89WZeVc5/g+ZuV853v/1cDVzlaQhbekZXVx7zve6tny/a2bxgtm0tkwaEGttmcTiBbMLykjV4OeselPVQlCS6t7k9tHFted8r2vL97tmFs6bxsdPPYJpba0EMK2tlY+feoSzSTYYP2fVm2qOEZSk+jd/ycBxVAAtraV4tXV3lrrpbekp/XE+f0ntxm4Vce0i3+smdPtL3s+c1R+hNZ7eEevLvbnzJe/nyALzalQL501ruoKgGZdSaMbPWfXLQlCShtNf/NS6KCpyIo+irl3Ue92k/vruWfzJ1nfxob06mRqPsjEP4pJtp7H67ln84OTdP1/1oahirH8phf5ZNPuXUgAslKqg0KK7yP+01JhE5k5L+9Wljo6O7OrqKjoNSRofl84pFWCDTZ4OH7yzca+tmpl5wTd3XtyX0iLAP7/4xFqnoyoYXIxBacxaLborHnPxLfQOMVvmtLZWfnDBa6p67WZT5Oe8038cQqknx0mXWQwWJCJWZ2bHSI51jKAkTURFTuThJCJNwRkOG1+R69q5lELtFLp+obMP1zULQUmaiIqcyMNJRGqroEXdneGw8RVZjPkfDbVTaNHtfxzWNQtBSZqI5i8pda+pVKuJU4q8drPp71a1ZQOQz47HrEEx6AyHja/IYsz/aKidQotu/+OwrjlZjCRNREVOnOKkLbUzXLeqGrzfznDY2BYvmD3k2LFaFGP991VRE9U002ylRX7OzF/Ctq+/n722P7UjtG3S/9/enYfJVZeJHv++ZiHNImEVSQeIk0wwhozRDnAnCKMIAXECIrIoYgQFFG5mxmeiOCpCRq4ZgsowcB3iBowMTORiDAoEBFzisDWECUsEwiLpBCUsQZEAIbz3j6oOlaaTVKf7dHVVfT/P00/V+Z3fqfN29clJvfXbhjHYLw7rgomgJA1UE46uXfJVy3M3E7tVqUC1TMY6z9/fCVgzzlZa06R77WQWrvkUf8+V62YfPv+1Y9lv7WSOKPzs6i1nDZVUP5yiWo3GGVqlPuVspf3L93vgcdZQSY2nhmOppMI4HlPqU85W2r98v+ubiaCk+uAU1WpEE44urbe17UggSo+uvyVtNmcr7V+1fL/nLVrO5Fk3M+qMnzF51s3MW7S88HM2GhNBSfXBsVRqUPPWTmbyyxcw6qXLmfzyBcxbO7nWITUsPzg2Pmcr7V+1er87x4IuX7Wa5PWxoP3xb7qR7iMmgpJ6pkZrnjlFtRpRLT/MNBvf6+bgsij9q1bv9+wFD643SyrA6jVrmb3gwULP22j3EWcNlVS9znF6nV00O8fpQfFd2Q48c/1zg2OpVPc29mHGD659y/e6ebgsSv+qxftdq7GJjXYfsUVQUvVqOU7PsVRqQE600H98r6XGUauxiY12H7FFUFL1aj1Oz7Xt1GB2Hd7S7dTrjT6xRS0W/G7W91pNosmWV5oxZex660VC/4xNbLT7iC2CkqrnOD01sFpMANCME1vUaoxNM77XahJNuLxSrcYmNtp9pNAF5SPiEOBfgUHAdzNzVpf9pwKnAWuBF4CTM/OB8r4vAieV903PzAUbO5cLykv9oOsYQSiN07OLpupcZ3LS9dvl/vhgUYvWsVqq5QLUzfZeq0l8a3w5Cexi25HwD/f1fzwNbqDfR3qyoHxhiWBEDAIeAg4COoA7geM6E71ynTdn5h/Lz6cCn83MQyJiHHAFsDewK/Bz4C8zcy0bYCIo9ZMm636i5lDL5KTZjDrjZ3T3ySOAx2Yd1t/hSPXvrOGwoX9VZ63q72hUYz1JBIscI7g3sDQzHy0HdSVwOLAuEexMAsu24vWr+HDgysx8GXgsIpaWX+/WAuOV6krNvpFynJ4aUKNNADCQNdoYG6nmtm3dQIugwza0cUWOERwBVF6VHeWy9UTEaRHxCHAuML0nx0rNqtHWsZFqrVYz0DWjRhtjI9XcgWeWhmlUcnklVaHIRDC6KXtDu3VmXpSZfwF8AfhyT46NiJMjoj0i2leuXNmrYKV6UquFVKVGNWPKWI4a+t8sHDqdR7f4KAuHTueoof9tclIAF/yW+pjLK2kzFdk1tAMYWbHdCqzYSP0rgW/35NjMnAPMgdIYwd4EK9UTu7FJfeuIQb/hg0O+y+C1LwHQGk8za9B3GTzorwA/TPU1F/yW+pjDNrQZimwRvBMYExGjImIocCwwv7JCRIyp2DwMeLj8fD5wbERsERGjgDHAHQXGKtUVu7FJfeymmeuSwE6D175UmhhJkqQGVFgimJmvAqcDC4AlwNzMvD8iZpZnCAU4PSLuj4h7gM8Bnygfez8wl9LEMtcDp21sxlCp2TjGRupjz3f0rFySpDpXZNdQMvNa4NouZWdWPP+7jRx7DnBOcdFJ9euIiSMYseynjLx7NjvnSp6KnVj2rhlMmnhIrUOT6pOz7prJdcQAACAASURBVEmSmkyRXUMlFWXxXCbd+1V2YSVvCtiFlUy696ulNf4k9Zyz7kmSmoyJoFSPbpoJa7pMDLNmteOZpM3lrHuSpCZTaNdQSQVxPJPU95x1T5LURGwRlOrRhsYtOZ5JkqTaWTwXvjUezhpeenTIhgYwE0GpHjmeSZKkgWXxXLhmenniqSw9XjPdZFADlomg1Bu1+ubP8UySJA0sjt9XnXGMoLS5Or/567zpd37zB/2TkDmeSZKkgcPx+6oztghKm8tv/iRJUifH76vOmAhKm8tv/iRJUifH76vOmAhKm8tv/iRJUifH76vOOEZQ2lwHnrn+GEHwmz9JkpqZ4/dVR2wRlDaX3/xJkiSpTtkiKPWG3/xJkiSpDtkiKEmSJElNxkRQkiRJkpqMiaAkSZIkNRnHCErqkXmLljN7wYOsWLWaXYe3MGPKWI6YOKLWYUmSJKkHTAQlVW3eouV88ep7Wb1mLQDLV63mi1ffC2AyKEmSVEcK7RoaEYdExIMRsTQizuhm/+ci4oGIWBwRN0XE7hX7zo2I+yNiSURcEBFRZKySNm32ggfXJYGdVq9Zy+wFD9YoIjWaeYuWM3nWzYw642dMnnUz8xYtr3VIkiQ1pMISwYgYBFwEHAqMA46LiHFdqi0C2jJzAnAVcG752L8GJgMTgPHAJOCAomKVVJ0Vq1b3qFzqic4W5+WrVpO83uJsMihJUt8rskVwb2BpZj6ama8AVwKHV1bIzFsy88Xy5m1Aa+cuYBgwFNgCGAL8ocBYJVVh1+EtPSqXesIWZ0mS+k+RieAIYFnFdke5bENOAq4DyMxbgVuAJ8s/CzJzSUFxSqrSjCljaRkyaL2yliGDmDFlbI0iUiOxxVmSpP5TZCLY3Zi+7LZixPFAGzC7vD0aeDulFsIRwPsiYv9ujjs5Itojon3lypV9Frik7h0xcQRfP3IvRgxvIYARw1v4+pF7OVGM+oQtzpIk9Z9NzhoaEacDl2fmcz187Q5gZMV2K7Cim9d/P/Al4IDMfLlc/CHgtsx8oVznOmBf4FeVx2bmHGAOQFtbW7dJppqDSxr0nyMmjvC9VSFmTBm73qy0YIuzJElFqaZFcBfgzoiYW54FtNrZO+8ExkTEqIgYChwLzK+sEBETgYuBqZn5VMWuJ4ADImJwRAyhNFGMXUPVLSeYkBqDLc6SJPWfyNx0Q1o5+TsY+CSlLpxzge9l5iObOO4DwPnAIOD7mXlORMwE2jNzfkT8HNiL0jhAgCcyc2p5xtH/C+xPqTvp9Zn5uY2dq62tLdvb2zf5u6g4tWqVmzzrZpZ3M4ZoxPAWfnPG+wo/vyRJkjQQRMRdmdlWTd2qFpTPzIyI3wO/B14FtgOuiogbM/PzGznuWuDaLmVnVjx//waOWwucUk1sGhhqudC4E0yoUdnlWZIkFWWTXUMjYnpE3EVpjb/fAHtl5meAdwMfLjg+1YlaTvu+6/AWpr5pIQuHTufRLT7KwqHTmfqmhU4wobpml2dJklSkaloEdwSOzMzfVRZm5msR8cFiwlK9qWWr3PnjHmb8Xd+lJV4BoDWe5l+GfJf7xu0B2DVU9WljX67YKihJknqrmslirgWe7dyIiG0iYh8A1/ZTp1pO+z7pkX9blwR2aolXmPTIvxV+bqkodnmWJElFqiYR/DbwQsX2n8tl0jo1XWj8+Y6elUt1wDX1JElSkapJBCMrphbNzNeocpIZNY+aTvu+bWvPyvvQvEXLmTzrZkad8TMmz7rZ8VvqMzX9ckWSJDW8ahK6RyNiOq+3An4WeLS4kFSvarbQ+IFnwjXTYU1Fl7khLaXyAtVyplQ1vs5ryFlDJUlSETa5jmBE7AxcQGnWjQRuAv6+ywLwNec6gk1u8Vy4aWapO+i2raUkcMLRhZ7S9QslSZI0kPTpOoLlhO/YXkclFWnC0YUnfl05mUfzcD0/SZLUaDaZCEbEMOAk4B3AsM7yzDyxwLikAW/X4S3dtgg6mUdjsQuwJElqRNVMFvMfwC7AFOCXQCvwpyKDkuqBk3k0h42t5ydJklSvqkkER2fmV4A/Z+alwGHAXsWGJQ18NZ0pVf3GLsCSJKkRVTNr6Jry46qIGA/8HtijsIikOlKzmVLVb+wCLEmSGlE1LYJzImI74MvAfOAB4F8KjUqSBgi7AEuSpEa00RbBiHgT8MfMfA74FfC2folKkgYI1/OTJEmNaKOJYGa+FhGnA3P7KR5JGnDsAixJkhpNNV1Db4yIf4yIkRGxfedP4ZFJkiRJkgpRzWQxnesFnlZRlthNVJIkSZLq0iYTwcwc1R+BSJIkSZL6xyYTwYg4obvyzLysimMPAf4VGAR8NzNnddn/OeBTwKvASuDEzPxded9uwHeBkZRaID+QmY9v6pySJEmSpI2rpmvopIrnw4ADgbuBjSaCETEIuAg4COgA7oyI+Zn5QEW1RUBbZr4YEZ8BzgWOKe+7DDgnM2+MiK2B16r5hSRJvTdv0XJnSpUkqYFV0zX0f1duR8S2wH9U8dp7A0sz89HycVcCh1Nah7DztW+pqH8bcHy57jhgcGbeWK73QhXnkyT1gXmLlvPFq+9l9Zq1ACxftZovXn0vgMmgJEkNoppZQ7t6ERhTRb0RwLKK7Y5y2YacBFxXfv6XwKqIuDoiFkXE7HILoySpYLMXPLguCey0es1aZi94sEYRSZKkvlbNGMFrKI3Rg1LiOI7q1hWMbsqymzIi4nigDTigIq73ABOBJ4D/AqYB3+ty3MnAyQC77bZbFSE1hzvnX8zIu2ezc67kqdiJZe+awaSpp9Q6LEl1YsWq1T0qlyRJ9aeaMYLnVTx/FfhdZnZUcVwHpYleOrUCK7pWioj3A18CDsjMlyuOXVTRrXQesC9dEsHMnAPMAWhra+s2yWw2d86/mPF3fZmWeAUCdmEl2971Ze4Ek0FJVdl1eAvLu0n6dh3eUoNoJElSEarpGvoEcHtm/jIzfwM8ExF7VHHcncCYiBgVEUOBY4H5lRUiYiJwMTA1M5/qcux2EbFTeft9VIwt1IaNvHt2KQms0BKvMPLu2TWKSFK9mTFlLC1D1u+N3zJkEDOmjK1RRJIkqa9Vkwj+iPVn7FxbLtuozHwVOB1YACwB5mbm/RExMyKmlqvNBrYGfhQR90TE/PKxa4F/BG6KiHspdTP9TpW/U1PbOVduoPzpfo5EUr06YuIIvn7kXowY3kIAI4a38PUj93KiGEmSGkg1XUMHZ+a6JqbMfKXcwrdJmXktcG2XsjMrnr9/I8feCEyo5jx63VOxE7vwxmTwqdiRXWoQj9SXXNKg/xwxcYTvrSRJDayaFsGVFS14RMThgM1LA9Syd81gda6fp6/OoSx714waRST1jc4lDZavWk3y+pIG8xYtr3VokiRJdaeaRPBU4J8i4omIeAL4AuCsIwPUpKmncN+7v8bv2YnXMvg9O3Hfu7/mRDGqey5pIEmS1HeqWVD+EWDfiNgaiMz8U/FhqTcmTT0FyonfLuWffrF4Ltw0E57vgG1b4cAzYcLR/XV2NTiXNJAkSeo7m2wRjIj/ExHDM/OFzPxTRGwXEV/rj+BURxbPhWumw/PLgCw9XjO9VC71gQ0tXeCSBpIkST1XTdfQQzNzVedGZj4HfKC4kFSXbpoJa7q0zKxZXSqX+oBLGkiSJPWdamYNHRQRW3Qu9h4RLcAWxYaluvN8R8/KpR7qnMHSWUMlSZJ6r5pE8IeU1vP7QXn7k8ClxYWkurRta7lbaDflUh9xSQNJkqS+scmuoZl5LvA14O3AOOB6YPeC41K9OfBMGNJlrNaQllK5JEmSpAGlmjGCAL8HXgM+DBwILCksItWnCUfD314A244EovT4txc4a6gkSZI0AG2wa2hE/CVwLHAc8AzwX5SWj3hvP8WmejPhaBM/SZIkqQ5sbIzgb4FfA3+bmUsBIuIf+iUqSZIkSVJhNtY19MOUuoTeEhHfiYgDgeifsCRJkiRJRdlgIpiZP87MY4A9gV8A/wC8JSK+HREH91N8kiRJkqQ+Vs2soX/OzMsz84NAK3APcEbhkUmSJEmSClHtrKEAZOazmXlxZr6vqIAkSZIkScXqUSIoSZIkSap/JoKSJEmS1GRMBCVJkiSpyZgISpIkSVKTKTQRjIhDIuLBiFgaEW+YaTQiPhcRD0TE4oi4KSJ277L/zRGxPCIuLDJOSZIkSWomhSWCETEIuAg4FBgHHBcR47pUWwS0ZeYE4Crg3C77/xn4ZVExSpIkSVIzKrJFcG9gaWY+mpmvAFcCh1dWyMxbMvPF8uZtlNYpBCAi3g28BbihwBglSZIkqekUmQiOAJZVbHeUyzbkJOA6gIh4E/ANYEZh0UmSJElSkxpc4GtHN2XZbcWI44E24IBy0WeBazNzWUR3L7PuuJOBkwF22223XgUrSZIkSc2iyESwAxhZsd0KrOhaKSLeD3wJOCAzXy4X/y/gPRHxWWBrYGhEvJCZ6004k5lzgDkAbW1t3SaZkiRJkqT1FZkI3gmMiYhRwHLgWOCjlRUiYiJwMXBIZj7VWZ6ZH6uoM43ShDJvmHVUkiRJktRzhY0RzMxXgdOBBcASYG5m3h8RMyNiarnabEotfj+KiHsiYn5R8UiSJEmSSiKzMXpUtrW1ZXt7e63DkCRJkqSaiIi7MrOtmrqFLigvSZIkSRp4TAQlSZIkqcmYCEqSJElSkzERlCRJkqQmYyIoSZIkSU3GRFCSJEmSmoyJoCRJkiQ1GRNBSZIkSWoyJoKSJEmS1GRMBCVJkiSpyZgISpIkSVKTMRGUJEmSpCZjIihJkiRJTcZEUJIkSZKajImgJEmSJDUZE0FJkiRJajImgpIkSZLUZEwEJUmSJKnJFJoIRsQhEfFgRCyNiDO62f+5iHggIhZHxE0RsXu5/J0RcWtE3F/ed0yRcUqSJElSMyksEYyIQcBFwKHAOOC4iBjXpdoioC0zJwBXAeeWy18ETsjMdwCHAOdHxPCiYpUkSZKkZlJki+DewNLMfDQzXwGuBA6vrJCZt2Tmi+XN24DWcvlDmflw+fkK4ClgpwJjlSRJkqSmUWQiOAJYVrHdUS7bkJOA67oWRsTewFDgkT6NTpIkSZKa1OACXzu6KctuK0YcD7QBB3QpfyvwH8AnMvO1bo47GTgZYLfdduttvJIkSZLUFIpsEewARlZstwIrulaKiPcDXwKmZubLFeVvBn4GfDkzb+vuBJk5JzPbMrNtp53sOSpJkiRJ1SgyEbwTGBMRoyJiKHAsML+yQkRMBC6mlAQ+VVE+FPgxcFlm/qjAGCVJkiSp6RSWCGbmq8DpwAJgCTA3M++PiJkRMbVcbTawNfCjiLgnIjoTxaOB/YFp5fJ7IuKdRcUqSZIkSc0kMrsdtld32trasr29vdZhSJIkSVJNRMRdmdlWTd1CF5SXJEmSJA08JoKSJEmS1GRMBCVJkiSpyZgISpIkSVKTMRGUJEmSpCZjIihJkiRJTcZEUJIkSZKajImgJEmSJDUZE0FJkiRJajImgpIkSZLUZEwEJUmSJKnJmAhKkiRJUpMxEZQkSZKkJmMiKEmSJElNxkRQkiRJkpqMiaAkSZIkNRkTQUmSJElqMiaCkiRJktRkCk0EI+KQiHgwIpZGxBnd7P9cRDwQEYsj4qaI2L1i3yci4uHyzyeKjFOSJEmSmklhiWBEDAIuAg4FxgHHRcS4LtUWAW2ZOQG4Cji3fOz2wFeBfYC9ga9GxHZFxSpJkiRJzaTIFsG9gaWZ+WhmvgJcCRxeWSEzb8nMF8ubtwGt5edTgBsz89nMfA64ETikwFj73uK58K3xcNbw0uPiubWOSJIkSZIAGFzga48AllVsd1Bq4duQk4DrNnLsiD6NrkiL58I102HN6tL288tK2wATjq5dXJIkSZJEsS2C0U1Zdlsx4nigDZjdk2Mj4uSIaI+I9pUrV252oH3uppmvJ4Gd1qwulUuSJElSjRWZCHYAIyu2W4EVXStFxPuBLwFTM/PlnhybmXMysy0z23baaac+C7zXnu/oWbkkSZIk9aMiE8E7gTERMSoihgLHAvMrK0TEROBiSkngUxW7FgAHR8R25UliDi6X1YdtW3tWLkmSJEn9qLBEMDNfBU6nlMAtAeZm5v0RMTMipparzQa2Bn4UEfdExPzysc8C/0wpmbwTmFkuqw8HnglDWtYvG9JSKpckSZKkGovMboft1Z22trZsb2+vdRivWzy3NCbw+Y5SS+CBZzpRjCRJkqTCRMRdmdlWTd0iZw1tbhOONvGTJEmSNCAVOUZQkiRJkjQAmQhKkiRJUpMxEZQkSZKkJmMiKEmSJElNxsliJEmS1C/WrFlDR0cHL730Uq1DkerasGHDaG1tZciQIZv9GiaCkiRJ6hcdHR1ss8027LHHHkRErcOR6lJm8swzz9DR0cGoUaM2+3XsGipJkqR+8dJLL7HDDjuYBEq9EBHssMMOvW5ZNxGUJElSvzEJlHqvL/4dmQhKkiSpaZx44onsvPPOjB8/foN1MpPp06czevRoJkyYwN13371u36WXXsqYMWMYM2YMl1566SbP94tf/IIPfvCDfRL75rjkkktYsWJFzc7f166//nrGjh3L6NGjmTVrVrd1vvnNbzJu3DgmTJjAgQceyO9+97t1+/z7vc5EUJIkSQPSvEXLmTzrZkad8TMmz7qZeYuW9/o1p02bxvXXX7/ROtdddx0PP/wwDz/8MHPmzOEzn/kMAM8++yxnn302t99+O3fccQdnn302zz333GbF8eqrr27WcT1V00Rw8Vz41ng4a3jpcfHcXr3c2rVrOe2007juuut44IEHuOKKK3jggQfeUG/ixIm0t7ezePFijjrqKD7/+c8D/v26MhGUJEnSgDNv0XK+ePW9LF+1mgSWr1rNF6++t9fJ4P7778/222+/0To/+clPOOGEE4gI9t13X1atWsWTTz7JggULOOigg9h+++3ZbrvtOOigg7pNKq+//nr23HNP9ttvP66++up15WeddRYnn3wyBx98MCeccAIvvfQSn/zkJ9lrr72YOHEit9xyC1D68H/44YdzyCGHMHbsWM4+++x1r/HNb36T8ePHM378eM4//3wAHn/88fVaOM877zzOOussrrrqKtrb2/nYxz7GO9/5TlavXt2r965HFs+Fa6bD88uALD1eM71XyeAdd9zB6NGjedvb3sbQoUM59thj+clPfvKGeu9973vZcsstAdh3333p6OgA8O/XhbOGSpIkacCZveBBVq9Zu17Z6jVrmb3gQY6YOKLQcy9fvpyRI0eu225tbWX58uUbLK/00ksv8elPf5qbb76Z0aNHc8wxx6y3/6677mLhwoW0tLTwjW98A4B7772X3/72txx88ME89NBDQCnpue+++9hyyy2ZNGkShx12GBHBD37wA26//XYyk3322YcDDjiA7bbbrtvf46ijjuLCCy/kvPPOo62trU/em6rdNBPWdElc1qwulU84erNesrv3//bbb9/oMd/73vc49NBDN3h8M//9bBGUJEnSgLNiVfetHxsq70uZ+YayiNhgeaXf/va3jBo1ijFjxhARHH/88evtnzp1Ki0tLQAsXLiQj3/84wDsueee7L777usSiYMOOogddtiBlpYWjjzySBYuXMjChQv50Ic+xFZbbcXWW2/NkUceya9//es++Z373PMdPSuvQjXvf6Uf/vCHtLe3M2PGjKqPb6a/n4lgQYro0y5JktQsdh3e0qPyvtTa2sqyZcvWbXd0dLDrrrtusLyrjSUnW2211brn3SUmG3qNDSWiAIMHD+a1115bt93bZQX6xLatPSuvQrXvP8DPf/5zzjnnHObPn88WW2zRo+Ob5e9nIliAovq0S5IkNYsZU8bSMmTQemUtQwYxY8rYws89depULrvsMjKT2267jW233Za3vvWtTJkyhRtuuIHnnnuO5557jhtuuIEpU6asd+yee+7JY489xiOPPALAFVdcscHz7L///lx++eUAPPTQQzzxxBOMHVv6/W688UaeffZZVq9ezbx585g8eTL7778/8+bN48UXX+TPf/4zP/7xj3nPe97DW97yFp566imeeeYZXn75ZX7605+uO8c222zDn/70p75+izbtwDNhSJekfUhLqXwzTZo0iYcffpjHHnuMV155hSuvvJKpU6e+od6iRYs45ZRTmD9/PjvvvPO6cv9+63OMYAFq2addkiSpEXR+Zpq94EFWrFrNrsNbmDFlbK8/Sx133HH84he/4Omnn6a1tZWzzz6bk046iX//938H4NRTT+UDH/gA1157LaNHj2bLLbfkBz/4AQDbb789X/nKV5g0aRIAZ5555hsmnhk2bBhz5szhsMMOY8cdd2S//fbjvvvu6zaWz372s5x66qnstddeDB48mEsuuWRd69V+++3Hxz/+cZYuXcpHP/rRdWPEpk2bxt577w3Apz71KSZOnLguln322YdRo0ax5557rjvHtGnTOPXUU2lpaeHWW29d162xcJ3jAG+aWeoOum1rKQnczPGBUGo5u/DCC5kyZQpr167lxBNP5B3veAdQ+v3b2tqYOnUqM2bM4IUXXuAjH/kIALvtthvz58/379dFbKxJs560tbVle3t7rcMAYNQZP6O7dzWAx2Yd1t/hSJIkDQhLlizh7W9/e63DGPAuueQS2tvbufDCC2sdijZDf/39uvv3FBF3ZWZVM8sU2jU0Ig6JiAcjYmlEnNHN/v0j4u6IeDUijuqy79yIuD8ilkTEBbGxzroDTC37tEuSJEnSphSWCEbEIOAi4FBgHHBcRIzrUu0JYBrwn12O/WtgMjABGA9MAg4oKta+Vss+7ZIkSapv06ZNszWwjtXL36/IMYJ7A0sz81GAiLgSOBx4oLNCZj5e3vdal2MTGAYMpdSjcgjwhwJj7VNF9WmXJEmSpL5QZCI4AlhWsd0B7FPNgZl5a0TcAjxJKRG8MDOX9H2IxTli4ggTP0mSpC4yc6PT80vatL6Y56XIMYLd/QuvKuKIGA28HWillFC+LyL276beyRHRHhHtK1eu7FWwkiRJKtawYcN45pln+uRDrNSsMpNnnnmGYcOG9ep1imwR7ABGVmy3AiuqPPZDwG2Z+QJARFwH7Av8qrJSZs4B5kBp1tDeBixJkqTitLa20tHRgV/gS70zbNgwWltbe/UaRSaCdwJjImIUsBw4Fvholcc+AXw6Ir5OqWXxAOD8QqKUJElSvxgyZAijRo2qdRiSKLBraGa+CpwOLACWAHMz8/6ImBkRUwEiYlJEdAAfAS6OiPvLh18FPALcC/wP8D+ZeU1RsUqSJElSM3FBeUmSJElqAANmQXlJkiRJ0sDTMC2CEbES+F2t4+jGjsDTtQ5CDc1rTEXy+lKRvL5UJK8vFWmgXl+7Z+ZO1VRsmERwoIqI9mqbZ6XN4TWmInl9qUheXyqS15eK1AjXl11DJUmSJKnJmAhKkiRJUpMxESzenFoHoIbnNaYieX2pSF5fKpLXl4pU99eXYwQlSZIkqcnYIihJkiRJTcZEsEARcUhEPBgRSyPijFrHo8YSEY9HxL0RcU9EtNc6HtW/iPh+RDwVEfdVlG0fETdGxMPlx+1qGaPq1waur7MiYnn5PnZPRHygljGqfkXEyIi4JSKWRMT9EfF35XLvYeq1jVxfdX0Ps2toQSJiEPAQcBDQAdwJHJeZD9Q0MDWMiHgcaMvMgbiGjepQROwPvABclpnjy2XnAs9m5qzyF1rbZeYXahmn6tMGrq+zgBcy87xaxqb6FxFvBd6amXdHxDbAXcARwDS8h6mXNnJ9HU0d38NsESzO3sDSzHw0M18BrgQOr3FMkrRBmfkr4NkuxYcDl5afX0rpPz6pxzZwfUl9IjOfzMy7y8//BCwBRuA9TH1gI9dXXTMRLM4IYFnFdgcNcMFoQEnghoi4KyJOrnUwalhvycwnofQfIbBzjeNR4zk9IhaXu47abU+9FhF7ABOB2/Eepj7W5fqCOr6HmQgWJ7opsx+u+tLkzHwXcChwWrnblSTVk28DfwG8E3gS+EZtw1G9i4itgf8H/H1m/rHW8aixdHN91fU9zESwOB3AyIrtVmBFjWJRA8rMFeXHp4AfU+qOLPW1P5THRnSOkXiqxvGogWTmHzJzbWa+BnwH72PqhYgYQulD+uWZeXW52HuY+kR311e938NMBItzJzAmIkZFxFDgWGB+jWNSg4iIrcqDlYmIrYCDgfs2fpS0WeYDnyg//wTwkxrGogbT+QG97EN4H9NmiogAvgcsycxvVuzyHqZe29D1Ve/3MGcNLVB5CtnzgUHA9zPznBqHpAYREW+j1AoIMBj4T68v9VZEXAH8DbAj8Afgq8A8YC6wG/AE8JHMdMIP9dgGrq+/odSlKoHHgVM6x3NJPRER+wG/Bu4FXisX/xOlcVzew9QrG7m+jqOO72EmgpIkSZLUZOwaKkmSJElNxkRQkiRJkpqMiaAkSZIkNRkTQUmSJElqMiaCkiRJktRkTAQlSeoiItZGxD0VP2f04WvvERF1tdaUJKnxDK51AJIkDUCrM/OdtQ5CkqSi2CIoSVKVIuLxiPiXiLij/DO6XL57RNwUEYvLj7uVy98SET+OiP8p//x1+aUGRcR3IuL+iLghIlpq9ktJkpqSiaAkSW/U0qVr6DEV+/6YmXsDFwLnl8suBC7LzAnA5cAF5fILgF9m5l8B7wLuL5ePAS7KzHcAq4APF/z7SJK0nsjMWscgSdKAEhEvZObW3ZQ/DrwvMx+NiCHA7zNzh4h4GnhrZq4plz+ZmTtGxEqgNTNfrniNPYAbM3NMefsLwJDM/Frxv5kkSSW2CEqS1DO5gecbqtOdlyuer8Ux+5KkfmYiKElSzxxT8Xhr+fl/A8eWn38MWFh+fhPwGYCIGBQRb+6vICVJ2hi/gZQk6Y1aIuKeiu3rM7NzCYktIuJ2Sl+mHlcumw58PyJmACuBT5bL/w6YJJhJnAAAAGtJREFUExEnUWr5+wzwZOHRS5K0CY4RlCSpSuUxgm2Z+XStY5EkqTfsGipJkiRJTcYWQUmSJElqMrYISpIkSVKTMRGUJEmSpCZjIihJkiRJTcZEUJIkSZKajImgJEmSJDUZE0FJkiRJajL/H6xESBxNIssiAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 1080x1080 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot train and validation accuracies of the two models\n",
    "\n",
    "train_accs = []\n",
    "val_accs = []\n",
    "for dropout in dropout_choices:\n",
    "  solver = solvers[dropout]\n",
    "  train_accs.append(solver.train_acc_history[-1])\n",
    "  val_accs.append(solver.val_acc_history[-1])\n",
    "\n",
    "plt.subplot(3, 1, 1)\n",
    "for dropout in dropout_choices:\n",
    "  plt.plot(solvers[dropout].train_acc_history, 'o', label='%.2f dropout' % dropout)\n",
    "plt.title('Train accuracy')\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend(ncol=2, loc='lower right')\n",
    "  \n",
    "plt.subplot(3, 1, 2)\n",
    "for dropout in dropout_choices:\n",
    "  plt.plot(solvers[dropout].val_acc_history, 'o', label='%.2f dropout' % dropout)\n",
    "plt.title('Val accuracy')\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend(ncol=2, loc='lower right')\n",
    "\n",
    "plt.gcf().set_size_inches(15, 15)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "pdf-inline"
    ]
   },
   "source": [
    "## Inline Question 2:\n",
    "Compare the validation and training accuracies with and without dropout -- what do your results suggest about dropout as a regularizer?\n",
    "\n",
    "## Answer:\n",
    "\n",
    "- 引入dropout层的网络过拟合现象得到缓解：\n",
    "  - 未使用dropout的网络`final_train_acc=0.978`, `final_val_acc=0.305`\n",
    "  - 使用dropout(p=0.2)的网络`final_train_acc=0.914`, `final_val_acc=0.327`\n",
    "\n",
    "- 适当设置p值或许能进一步缓解过拟合现象\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "pdf-inline"
    ]
   },
   "source": [
    "## Inline Question 3:\n",
    "Suppose we are training a deep fully-connected network for image classification, with dropout after hidden layers (parameterized by keep probability p). If we are concerned about overfitting, how should we modify p (if at all) when we decide to decrease the size of the hidden layers (that is, the number of nodes in each layer)?\n",
    "\n",
    "## Answer:\n",
    "\n",
    "- 考虑设p的数量级为样本数量级倒数，在此数量级对p微调\n",
    "- p=0.01，可以做到`final_train_acc=0.392`, `final_val_acc=0.267`（val_acc和train_acc更接近）\n",
    "- p=0.05，可以做到`final_train_acc=0.648`, `final_val_acc=0.299`（val_acc更接近p=1时的）\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
